Distributed preconfiguration of spare capacity in closed paths for network restoration

ABSTRACT

A method for restoring traffic in a network. The network includes plural distinct nodes interconnected by plural distinct spans, each span having working links and spare links. Each node has a digital cross-connect switch for making and breaking connections between adjacent spans forming span pairs at a node. Cross-connections between spare links in adjacent spans are made such that sets of successive nodes through which the adjacent spans form span paths form closed paths. A method of finding and construction closed paths is described in which statelets are broadcast through the network. In a preferred method of implementation of the method, the statelet broadcast occurs not in response to a network failure, but across the entire network before any particular span failure and may be carried out during normal network operations as a continual re-configuration of the network.

BACKGROUND

1. Technical Field

This patent document relates to methods for restoring networks of spans,for example telecommunications networks, upon the occurrence of afailure of one of the spans or nodes of the network.

2. Description of the Related Art

Methods for the restoration of digital transport networks have beenstudied for several years. Some researchers have concluded thatdistributed real-time protocols may operate too slowly to meetoperational objectives, such as the 2 second call-dropping threshold invoice networks. This problem can be overcome by preparing the networkfor anticipated failures by optimally preconfiguring resources whichwill be used to react to the failure. This places the network into astate of “best readiness” to face a given set of anticipated failures.See W. D. Grover and M. H. MacGregor, “Potential for spare capacitypreconnection to reduce crossconnection workloads in mesh-restorablenetworks”, Electronics Letters, Vol. 30, No. 3, pp. 194-195, Feb. 3,1994. By contrast, M. Herzberg and S. J. Bye, “An optimal spare-capacityassignment model for survivable networks with hop limits”, Proceedingsof IEEE Globecom '94, Vol. 3, pp. 1601-1606, IEEE, 1994 deals with theproblem of optimum spare capacity amount determination, but does notpre-configure such spares.

The tremendous interest in real-time network restoration over the lastfew years is evidence both of the importance and difficulty of thisproblem. Evidence has been mounting that it may not always be possibleto meet operational requirements with real-time techniques. Thesuggested cures have ranged from high speed, parallel computingarchitectures for digital crossconnect machines to simplified networkarchitectures where restoration can be made to occur more quickly inreal-time. The proposal of Grover and MacGregor suggests anticipatingfailures, and preconfiguring the network to handle them so thatreal-time computation or reaction delay (one or both) is not requiredexcept at the endpoints of the failure. Thus, preconfiguration methodsapply where computing a reaction or implementing it (or both) is/are toolengthy a process.

One of the inventors has previously disclosed, in Canadian patentapplication No. 2,161,847 published May 1, 1997, a method for restoringtraffic in a network in which the network is pre-configured for spanrestoration. The network includes plural distinct nodes interconnectedby plural distinct spans, each span having working links and sparelinks. Each node has a digital cross-connect switch for making andbreaking connections between links in adjacent spans forming paths orpath segments (span pairs) through nodes. In a broad characterization ofthe method, there are three steps.

Step 1: For each of at least two possible span failures, (a) find thenumber of restoration routes available in case of the occurrence of eachspan failure, (b) determine the resources used by each restorationroute, and (c) determine the amount of flow to be restored for each spanfailure.

Step 2: find, in a computer, the amount of flow f^(p) to be restoredalong each restoration route that minimizes total unrestored flow forall possible span failures identified in step 1.

Step 3: form connections at each digital cross-connect switch in thenetwork along each restoration route before occurrence of one of thepossible span failures identified in step 1 to permit the amount of flowf^(p) determined in step 2 to be carried by each respective restorationroute upon the occurrence of one of the possible span failuresidentified in step 1.

PC-restoration design may be generated using trees and unconstrainedpatterns of genetic algorithm techniques. These approaches have apractical drawback in that there is no constraint on the nodal degreesthat a pattern can have (other than the degrees of the underlyingnodes.)

The inventor has also proposed, in U.S. Pat. No. 4,956,835, issued Sep.11, 1990, a method and apparatus of restoring communications between apair of nodes in a network having an arbitrary number of nodes and anarbitrary number of spans interconnecting the nodes. Each span hasworking circuits between nodes designated for transmitting actualcommunications traffic and spare circuits capable of, but not designatedfor, transmitting actual communications traffic. The method comprisesthe steps of (a) establishing one or more independent communicationpaths between the pair of nodes through a series of spare circuits ofspans interconnecting the pair of nodes and other interconnected nodesin the network; and (b) redirecting communications traffic intended forone or more failed spans interconnecting the pair of nodes through oneor more of the paths. However, self-healing networks operating pursuantto the method described in U.S. Pat. No. 4,956,835 are reactive in thatthe establishment of a communications path occurs only after theoccurrence of a span failure, or in response to a demand for additionalcapacity. In addition, the self-healing network utilizes distinct endnodes for the initiation of a broadcast of signatures and initiatingconstruction of a restoration path. Further, the self-healing networkutilizes the broadcast of signatures that in themselves do not containthe information required to construct the restoration path.

There also has been proposed, in order to enhance survivability ofnetworks, the concept of self-healing rings (Wu, “Fiber Network ServiceSurvivability”, Boston, USA, 1992, in particular, Ch. 4). In aself-healing ring, nodes are interconnected by spans organized intorings. Each span includes working links and spare links. Add-dropmultiplexers (ADMs) at the nodes react to a span failure on the ring torestore a communications path in one of two ways. If only a working linkis lost, communications may be restored along the same span by droppingthe working link and adding a spare link on that span. If an entire spanis lost, communications may be restored by re-routing traffic in theopposite direction around the ring, using either working or spare linksin the ring.

Self-healing rings only protect the spans on the ring itself and provideat most one restoration route per failure. In addition, protection pathsof one ring overlapping a span are only available to failed workingspans in the same ring. A working path routing must be realized by asuccession of ring-arc traversals and ring-to-ring transfers. Each ringfunctions on its own in its as built configuration.

BRIEF SUMMARY

There is proposed a method that overcomes disadvantages in theseaforementioned network restoration schemes. Therefore, in accordancewith an embodiment, there is proposed a method of operating atelecommunications network in which the telecommunications networkincludes plural distinct nodes interconnected by plural distinct spans,each node having a digital cross-connect switch for making and breakingconnections between links in adjacent spans forming span paths throughthe node, the method comprising the steps of:

-   -   a) providing a set of successive nodes capable of forming a        closed path in the network, with at least one spare link between        each pair of adjacent nodes in the closed path; and    -   b) forming a cross-connection at each node in the closed path to        connect spare links in each of the adjacent spans lying in the        closed path and thus form a span path through each node in the        closed path.

By configuration of the controller of the digital cross-connect switchat a node, upon occurrence of a span failure on any span between any twonodes in the closed path, wherein the span failure is not at a span inthe closed path, telecommunications traffic may be routed along theclosed path.

The closed path preconfiguration method greatly simplifies therestoration protocol as only the end nodes of a failed span need to actto substitute traffic; no signalling is required to or amongst othernetwork nodes. Restoration becomes even simpler than in thebidirectional line switched ring (BLSR) signalling protocol. Closed pathpre-configuration is also different from the idea of self-healing rings(SHRs). The preconfigured closed paths here may contribute flexibly tothe restoration of many network spans both on and off of the closedpath. This is fundamentally different than SHR technology: SHRs onlyprotect the spans on the ring itself and provide at most one restorationroute per failure. However, PC closed paths within the spare capacity ofa mesh network contribute up to two restoration paths per failure andcan contribute to restoration of any network span failure not just thosealong the closed path. It is in that sense that this remains a mesh typerestoration technology. The capacity requirements also reflect the verysignificant difference between PC-closed path technology and SHRtechnology since here we see closed path PC designs in the same capacityregime as span restorable mesh design whereas corresponding SHR basednetworks are typically 3 to 6 times greater in total capacity (W. D.Grover, Chapter 11 of Telecommunications Network Management Into the21st Century, IEEE Press, 1994, pp. 337-413, edited by S. Aidarous andT. Plevyak). In sum, closed path-oriented preconfigured spare capacitydesign (DCPC) seems to open the door to a restoration technology thatenables the restoration speed of rings while retaining the capacityefficiency of a span restorable mesh network.

These and other embodiments are described in the detailed descriptionthat follows.

BRIEF DESCRIPTION OF THE DRAWINGS

There will now be described preferred embodiments, with reference to thedrawings, by way of illustration, in which like numerals denote likeelements and in which:

FIG. 1A is a schematic of a network showing an exemplary pre-configuredpath that a single preconfigured closed path can provide towards therestoration of a failed span;

FIG. 1B is a schematic of the network of 1A, with differentcross-connections, showing two preconfigured paths that a singlepreconfigured closed path can provide towards the restoration of afailed span;

FIGS. 2A-2D show examples of patterns generated in a test network usinga linear programming based preconfiguration closed path;

FIG. 3 shows a node of a network with a single incoming stateletarriving on one link of a span represented as having two links, whereinthe node is of degree three and has two other spans, one of which has asingle spare link and the other of which has two spare links;

FIG. 4 shows the node of FIG. 3 broadcast to the largest extentpossible;

FIG. 5 shows the node of FIG. 3 with five incoming statelets;

FIGS. 6A-6E show respectively the broadcast of statelets s₁-s₅ in orderof relative ranking of the score of the statelets;

FIG. 7 shows the resulting overall statelet broadcast pattern for thenode shown in FIGS. 3-6E;

FIG. 8A shows a node with a single incoming statelet s₁ belonging toindex m;

FIG. 8B shows the node of FIG. 8A with a second statelet s₂ arriving atthe node, in which the statelet s₂ belongs to the same index m as thestatelet s₁ and has a better score;

FIG. 8C shows the node of FIGS. 8A and 8B with s₁'s statelet broadcastshifted to s₂;

FIG. 9 shows an example of a tandem or intermediate node stateletbroadcast, illustrating that only simple closed paths are generatedwhich visit an originating node only once;

FIGS. 10A-10F illustrate how a tandem node evaluates and updates theincremental preconfigured path count of an incoming statelet for anetwork with nodes S, 1, 2, T and 3;

FIG. 11 illustrates the evaluation of the total PC path count by thetandem nodes along a statelet broadcast route;

FIG. 12 is a graph showing the best received incoming statelet score atan originating node plotted against a simulation iteration when usingDCPC with node order in order of decreasing working links in a samplenetwork (Net1);

FIGS. 13A-13H illustrate a network with an originating node shown inbold, illustrating the successive displacement of the best scoringclosed path candidate contained in received incoming statelets at anoriginating node;

FIGS. 14A-14D show examples of the patterns which result in a samplenetwork using the DCPC with originating node in descending node totalworking links;

FIGS. 15A-15E show an example of patterns which result in a samplenetwork using the DCPC with originating node in increasing node totalworking links;

FIGS. 16A-16F show examples of patterns which result in a sample networkusing the DCPC with closed path order in descending node total workinglinks with sparing patterns generated by IP2-closed path (definedbelow);

FIGS. 17A-17E are examples of patterns which result in a sample networkusing the DCPC with closed path order in increasing node total workinglinks with the sparing plans generated by IP2-closed path;

FIG. 18 is a schematic illustrating a digital cross-connect machineequipped with the necessary elements so that when each node of thenetworks for example of FIGS. 1A-2D is so equipped, the objective ofpre-configuration is achieved;

FIG. 19 is a schematic illustrating signature reception and transmissioncircuits added by a preferred embodiment to each transmission lineinterface block illustrated in FIG. 3;

FIGS. 20A-20C illustrate the content and organization of information inthe signature registers shown in FIG. 19 according to a preferredembodiment; and

FIG. 21 illustrates the highest level logical structure of a FiniteState Machine used at a node for implementation.

DETAILED DESCRIPTION Definitions

The term link denotes an individual bidirectional digital signal carrierbetween adjacent nodes at the DCS signal management level, for instanceone DS-3, STS-1 or STS-N. A link between two nodes may be one of manylinks between the two nodes. A span is the set of all working and sparelinks in parallel between adjacent nodes. A preconfiguredcrossconnection is a crossconnection at a node which is preset betweenspare links in statistical anticipation of a span failure. A PC plan isthe network wide state of all preconfigured crossconnections and thespare links which they interconnect. An individual PC pattern (or just“pattern”) is a connected subgraph of unit link capacity formed fromspare links and preconfigured crossconnections. An individual patternhas at most one link on any span. A useful path, or useful PC path, is apath contained in a PC pattern which is able to contribute to therestoration of a given failure. Any PC paths in a PC plan in excess ofthe number of working links to be restored in given failure are notuseful paths. An uncovered working link is a working link which does nothave a PC restoration path immediately available to it in the PC planshould it fail. This does not mean it is unrestorable. It may still berestorable using on-demand crossconnect operations, to supplement theuseful paths in the PC plan for the given failure. Related to this arethe concept of ksp restorability and two-step restorability. Thek-shortest paths (KSP) restorability of a network is the restorabilitywhen using a KSP algorithm to calculate a restoration pathset for eachof the networks failed spans with the given numbers of spare links butwithout regard to any PC plan. The preconfigured (PC) restorability of anetwork is the restorability when relying only on restoration pathsegments that are found already formed with a network's PC plan. The PCrestorability indicates how effective the network's PC plan is atproviding immediately useful paths for restoration without any newcrossconnection workload. The “two-step” restorability of a network isthe total network restorability when all useful PC paths are firstapplied to the span failure and, if needed, additional on-demand (2ndstep) KSP paths are found within the unallocated spare capacity of thePC plan plus any pruned spare capacity from the PC restoration step. Anunallocated spare link is a spare link which does not belong to anyindividual PC pattern of the PC plan. A pruned spare link is a sparelink which was a part of a PC pattern but was not needed to form a PCpath for a given span failure and is “pruned” to isolate it from theuseful portion(s) of a PC pattern.

Network Design

Each node N in the networks shown in the figures incorporates aconventional digital cross-connect switch (DCS) for making and breakingconnections, between links LXY and LZX in adjacent spans meeting at thenode. The links may connect to the nodes through fiber optic terminals,for example AT&T FT Series G, for electrical/optic conversion of thesignals. Exemplary digital cross-connect switches are the AT&T DACSIV-2000, Alcatel RDX-33 and the Tellabs Titan Model 5533. Adjacentspans, for example spans including links LYX and LXZ at a node NX form aspan pair. A network operations center (not shown), for example an AT&TTransport and Maintenance System, controls the operation of the digitalcross-connect switches at the nodes via a control overlay of for exampleX.25 communications links (not shown). The network operations centerincludes a computer that may instruct the digital cross-connect switchesto make and break connections between links in adjacent spans. Thecomputer may be, for example, a UNIX work station that may be connecteddirectly to the digital cross-connect switches via the X.25communications links and through an X.25 packet assembler-disassembler.

According to an embodiment, a network is pre-configured with nodesconnected in closed paths without intervention of the network operationscenter. The method of pre-configuration operates independently at eachnode to achieve an overall network configuration of spare links. Aclosed path is a set of successive nodes in a network interconnected bysuccessive spans, in which each successive span includes at least onespare link and spare links in adjacent spans meeting at a node areconnected by the DCS at the node to form a span path through the node.At each node in a closed path there is one and only one span paththrough the node that is connected within the closed path. In otherwords, all nodes in a closed path are of degree two. A node may havemore than one closed path passing through the node, but the closed pathsare not connected to allow communications traffic between them.

Exemplary closed paths are shown in FIGS. 1A and 1B, in which are showntwo networks formed of plural distinct nodes N1, N2, N3 and N4interconnected by plural distinct links L12, L23, L13, L34 and L14,wherein LXY indicates a spare link between node NX and node NY. In FIG.1A, cross-connections C412, C124 and C142 are made to connect nodes N1,N2 and N4 through spare links L12, L14 and L24 in a closed path. In FIG.1B, cross-connections C312, C124, C243 and C134 are made to connectnodes N1, N2, N3 and N4 through links L12, L24, L34 and L13.

Various methods may be used to generate connected closed pathspre-configured for use in restoration of networks. Integer LinearProgramming (IP) may be used to generate preconfiguration plans usingonly closed paths as the building block element.

A. IP-1: PC Design within a Given Spare Capacity Plan

The method that follows is a modified version of an approach used togenerate preconfigured restoration plans using linear segments in M. H.MacGregor, W. D. Grover, K. Ryhorchuk, “Optimal spare capacitypreconfiguration for faster restoration of mesh networks”, accepted forpublication by Journal of Network and Systems Management. The embodimentdisclosed here permits a set of pre-defined arbitrary example patternsto be defined as building blocks on which it can draw to generate amaximally preconfigured PC design. The result is optimal only withregard to the building block patterns in the input set. Afterintroducing the general formulation we use it specifically to generatepure closed path-oriented PC designs.

For a set P consisting of N_(p) elemental types of building-blockpatterns, in which each of the basic patterns has a topologicallydistinct configuration on the network graph, and where the network sparecapacity is already given, the following formulation optimizes the PCdesign within the given set of existing spares. The objective functionis:

$\begin{matrix}{{{minimize}\mspace{14mu} {\sum\limits_{j = 1}^{s}\; {u_{j}\mspace{14mu} {subject}\mspace{14mu} {to}}}},} & {{EQ}\mspace{14mu} 1} \\{s_{j} \succcurlyeq {\sum\limits_{i = 1}^{N_{p}}\; {\left( {spc}_{i,j} \right)n_{i}}}} & {{EQ}\mspace{14mu} 2} \\{{u_{j} + {\sum\limits_{i = 1}^{N_{p}}\; {\left( {pc}_{i,j} \right)n_{i}}}} = {w_{j} + r_{j}}} & {{EQ}\mspace{14mu} 3} \\{0u_{j}w_{j}} & {{EQ}\mspace{14mu} 4} \\{{r_{j} \succcurlyeq 0}{{{\forall j} = 1},2,\ldots \mspace{14mu},S}{and}} & {{EQ}\mspace{14mu} 5} \\{{n_{i} \succcurlyeq 0}{{{\forall i} = 1},2,\ldots \mspace{14mu},N_{p}}} & {{EQ}\mspace{14mu} 6}\end{matrix}$

where: u_(j) is the number of working links on span j which do not havea PC path available to them (i.e., ‘unprotected’ working links), S isthe number of network spans, s_(j) is the total number of spare links onspan j, n_(i) is the number of copies of pattern i in the PC design,spc_(i,j) is the number of spare links required on span j for one copyof pattern i, pc_(i,j) is the number of preconfigured paths that asingle copy of pattern i from pattern set P can provide for failed spanj, w_(j) is the number of working links which are on span j, and r_(j)is the number of PC paths provided in the design which are in excess ofthe working links on span j.

Eq. 2 constrains the total number of spare links in PC patterns to thenumber of spare links which are present on each span. Eq. 3 gives therelationship between the number of working links on each span (w_(j)),the number of PC paths provided by the PC plan for protection of workinglinks on that span, the number of unprotected working links on each span(u_(j)), and the number of PC paths which may be provided in excess ofthe protection required for span j(r_(j)). Note that either one of u_(j)or r_(j) is implicitly zero (a network span cannot both be overprotectedand underprotected with PC paths.) The design that is returned from thistableau is represented in the n_(i) variables from which the completenetwork PC plan can be built.

B. IP-2: PC Design with Spare Capacity Placement

An IP may be formulated for preconfigured spare capacity design in whichthe spare capacity quantities are free to be determined in conjunctionwith the preconfigured pattern design so as to reach a min-capacityresult that is assured to be 100% PC-restorable. As in the previoussection, a design is formed from a pre-defined set of example patternson the network graph.

The formulation is as follows:

$\begin{matrix}{{minimize}\mspace{14mu} {\sum\limits_{j = 1}^{s}\; {c_{j}s_{j}\mspace{14mu} {subject}\mspace{14mu} {to}\text{:}}}} & {{EQ}\mspace{14mu} 7} \\{s_{j} \succcurlyeq {\sum\limits_{i = 1}^{N_{p}}\; {\left( {spc}_{i,j} \right)n_{i}}}} & {{EQ}\mspace{14mu} 8} \\{{\sum\limits_{i = 1}^{N_{p}}\; {\left( {pc}_{i,j} \right)n_{i}}} = {w_{j} + r_{j}}} & {{EQ}\mspace{14mu} 9} \\{{r_{j} \succcurlyeq 0}{{{\forall j} = 1},2,\ldots \mspace{14mu},S}} & {{EQ}\mspace{14mu} 10} \\{{n_{i} \succcurlyeq 0}{{{\forall i} = 1},2,\ldots \mspace{14mu},N_{p}}} & {{EQ}\mspace{14mu} 11}\end{matrix}$

where c_(j) is the cost or length of span j. The PC design generatedwith this IP is able to fully restore any failed network span using onlyPC paths. Other variables are as in the prior IP. Note that since thisIP generates the s_(j) values, the prior Eq. 2 changes now to anequality (in Eq. 8) from an inequality.

C. Adaptation of the General Formulation for Closed Path-Oriented PC-IPDesigns

In this section, the first general IP formulation given above is usedwith a set P containing only distinct closed paths of the network graph.The constraints and objective function required for use in an IP aregiven by Eqs. 1 to 6. However, it is required that the coefficientspc_(i,j) and spc_(i,j) be evaluated for the closed paths contained inthe pattern set. The coefficients w_(i) and s_(i) are set by the networkworking and spare capacity placement plan. spc_(i,j) is equal to thenumber of spare links required on span j to build a single copy ofpattern i. For a closed path i in P spc_(i,j) can be either 0, 1 or 2,as illustrated in FIGS. 1A and 1B. It is zero if either of the failedspan end nodes are not on the closed path (e.g., for span L34). It isone if both of the span end nodes are on the closed path and the endnodes of the failure are also adjacent to one another along the closedpath (FIG. 1A). pc_(i,j) is two if both of the failure span end nodesare on the closed path but are not adjacent to one another on the closedpath (FIG. 1B).

The second general IP method given above to generate the minimal sparingfor 100% PC restorability can also be used with a set P containing onlyclosed paths. The constraints and objective function are given by Eqs. 7to 11 and the coefficients pc_(i,j) and spc_(i,j) are evaluated inadvance as above for the closed paths contained in pattern set P.

D. Results of IP-1 and IP-2 Closed Path-Oriented PC Designs

The set P used in the test networks, for both of the closedpath-oriented IP design approaches, included all distinct closed pathsup to a maximum hop length of infinity for test networks 1, 2 and 3. Fornetworks 4 and 5, the maximum closed path length was 12 and 25 hopsrespectively. This means that for networks 1-3 the complete set of allpossible closed paths was included in the designs. For networks 4 and 5,the limiting closed path lengths are still quite high; certainly at orabove practical limits that we might expect for PC deployment. The IPswere run to completion for test networks 1, 2 and 3 with CPLEX 3.0(CPLEX Optimization Inc., Using the CPLEX Callable Library—V3.0, Suite279, 930 Tahoe Blvd., Bldg. 802, Incline Village, Nev. 89451), and so,for these test networks, the result is optimal. In networks 4 and 5, theCPLEX runs terminated before true completion of the IP engine becausethey ran out of memory. The design solutions returned are, therefore,the best feasible designs found prior to the point of memory exhaustionby CPLEX.

An example of the closed path-oriented PC patterns generated in testnetwork 1 is shown in FIGS. 2A-2D. The restorability performance withclosed path-oriented PC design, including the two-step restorability inthe existing spare capacity (IP-1), is given in Table 1. There is nocorresponding table for IP-2 closed path PC design as every network is100% PC restorable and there is no case of two-step restorability. Forthe design of PC closed path plans within an existing KSP sparing plan(IP-1), the PC restorability ranged from 75% to 97% in the testnetworks. The low of 75% in network 4 may be due to the relativelyrestricted closed path set and partial CPLEX termination result. A hoplimit of 12 was necessary to control the size of the closed path-set (P)in the presence of the high average nodal degree of this network. Theremaining networks had PC restorabilities in the range of 94% to 97%.The overall restorability, using PC restoration first, followed by2nd-step KSP restoration, was from 94% to 99%.

The comparison of crossconnection workloads in IP-1 and IP-2 designs isin Table 2. With the IP-1 designs, there is still a mixture ofcrosspoint assertion and crosspoint opening actions required althoughthe total crossconnection make workload is 76% to 100% lower than thetotal crossconnect closures required for KSP restoration. In the secondhalf of Table 2, however, we see the remarkable operational propertyimplied by the IP-2 type designs: 100% restorability is achieved solelythrough unmaking connections to isolate the desired restoration pathsegments. There is also an unavoidable minimum action of making onecrossconnection per demand unit to substitute traffic into therestoration paths at the failure end nodes (these traffic substitutioncrossconnections are fundamentally common to all possible restorationschemes).

Table 3 shows the capacity cost of this type of design (IP-2) relativeto the KSP baseline. Table 3 shows that the IP-2 designs appear to stillbe highly capacity efficient. The total extra spare capacity to support100% restorability through preconfigured closed paths ranged from zeroto 18.5%. Although again, the largest increase in sparing requirementoccurred for Net4 where the CPLEX design did not reach its optimaltermination. In practice, aside from Net4, these results suggest thatwith closed path-oriented preconfiguration of spare capacity, 100% ofthe required restoration paths can be preconfigured in advance offailures, often without significantly more total sparing than aconventional span-restorable network requires.

Implementation of Closed Path Pre-Configured Network

A distributed preconfigured closed path design (DCPC) algorithm, whichcan be run in a network's nodes, is presented. The algorithm isdistributed in the sense that its execution is spread amongst thesignificant processing power present in the DCS machines which form amesh network's nodes. A network's nodes can execute the algorithm usingonly the information which is available locally at each node. Thisalgorithm is a heuristic so the results are not guaranteed to be optimalbut, for the advantage of being to design a PC plan using only theprocessing power present within the network, sub-optimal designs may beacceptable.

This DCPC algorithm is based on the self-healing network (SHN) protocoldescribed in U.S. Pat. No. 4,956,835 issued Sep. 11, 1990. Both the SHNprotocol and the DCPC method implement a distributed algorithm whichgenerates efficient path sets for span restoration and both use stateletbased processing. A statelet is embedded on each network link (as forexample link L1 in FIG. 3) and contains the current state of the link asdescribed by a number of fields. A link is a bidirectional signalcarrier and, so each link, viewed locally by a node N (FIG. 3), may haveboth an incoming statelet and outgoing statelet. As shown in FIG. 3, anincoming statelet arrives at a node N on a link L1, and originates fromthe adjacent node connected to the receiving node through the link. Anoutgoing statelet s_(o) is broadcast from a node, such as node N in FIG.3, through a link, such as link L3 (FIG. 4), and arrives at the adjacentnode which is connected to the broadcasting node through the link.

Each outgoing statelet has, in general, an incoming statelet which formsits precursor. s_(o) in FIG. 3 is an outgoing statelet for which s₁ isthe precursor. An incoming statelet forms a precursor to an outgoingstatelet if the incoming statelet was broadcast to form the outgoingstatelet. In other words, an incoming statelet which forms the precursorof an outgoing statelet is the source for the outgoing statelet. Anincoming statelet can be the precursor for many outgoing statelets butan outgoing statelet can have only one precursor incoming statelet.

As a statelet is broadcast throughout a network, it forms a stateletbroadcast tree which, at each node in the tree, is rooted at theprecursor incoming statelet from which the outgoing statelets arepropagated. The particular route which a statelet travelled from thesender node to its present location is called the statelet broadcastroute.

The distributed algorithm operates by the application of a set of rulesat each node which preferentially broadcast certain incoming stateletsover others. Each node acts only on the basis of information availableto it but the interaction of the nodes' actions results in thegeneration of a solution which is efficient at a global level.

Statelet Format

The statelet format used in the DCPC algorithm has 6 main fields asshown in FIGS. 20A-20C:

node ID field 110 and 112: There is only one basic format of restorationsignature, but it is interpreted slightly differently depending onwhether it is a “transmit signature” or a “receive signature”. Everytransmit signature becomes a receive signature at the opposite end ofthe link on which it is transmitted. Each port of nodes in the networkhas provision for one transmit signature and one receive signature. Thefield NID (Node Identifier) is written to a transmit signature by thenode sending that signature and contains the network-wide identifier ofthe node originating the signature. This NID field appears in the NIDfield 110 of the corresponding receive signature at an adjacent node.The NID field is used so that each DCS machine can recognize thegrouping of links arriving at its site into logical spans to the sameadjacent node by associating all links with equal receive signature NIDfields into a logical span for control logic purposes.

An alternative implementation is to store data at each DCS about thefacility spans terminating at its site in which case NID's are notneeded. However, the NID-based embodiment is preferred because it isconsistent with an objective that all information needed by each node toperform the method comes to that node through the network connectionsthemselves, thereby continually being up-to-date and eliminating thesignificant practical problems of maintaining accurate real-timedistributed network configuration databases at each node.

index field 114: Each statelet belongs to an index family. At a node,when an outgoing statelet is broadcast from an incoming statelet, theoutgoing statelet's index field is set equal to the index field of theincoming statelet from which it was broadcast.

hopcount field 116: As a statelet is relayed from node to node, a countof the number of hops it has taken is maintained. The hopcount in anincoming statelet is equal to the number of times that the statelet hasbeen relayed by previous nodes to reach the node at which it hasarrived.

sendNode field 118: All statelet broadcasts originate at a single nodewhich acts as the sender node for the network. The sendNode field is setequal to the name of the sender node, and is the same for all stateletsin the network.

numPaths field 120: As a statelet is relayed through the network, thenumber of useful PC paths which the statelet could provide if it couldsuccessfully loop back to the sender node to form a closed path, isevaluated. This field, in an incoming statelet, is set equal to thevalue which was evaluated at the node from which the incoming stateletoriginated.

route field 122: This field contains the route (sequence of successivenodes), from the originating node, which a statelet broadcast followedto become an incoming statelet at a particular node. This field ispreferably an appended list of the nodes in order that the stateletvisited the nodes. Alternatively, and equivalently, the field may be avector with a size equal to the number of nodes in the network. Eachnode in the network is represented by a cell in the route vector. Eachcell corresponding to a node contains the name of the node which was thepredecessor of the node along the route. That is, the cell correspondingto a node points back to the node which fell before the cell node on theroute. If a node does not fall on the route then its cell is set equalto NIL. As a node broadcasts a statelet to an adjacent node, the routefield is updated so that the receiving node's cell points to thebroadcasting node. Using the route vector allows the route, which wastaken by a statelet broadcast to reach a particular node, to be tracedback to the sender node.

These five fields form the basis of the statelet used in the distributedPC closed path design method.

Broadcast Protocol

In the SHN protocol there are three node roles: that of the sender, thechooser and the tandem nodes. The sender node is the source of thebroadcast pattern and the chooser node is the recipient of the broadcastpattern. The tandem nodes in the network relay received stateletsaccording to the rules of the SHN protocol. The SHN protocol isconcerned with the restoration of a span failure at the span level. Theend nodes on a failed span become the sender and chooser nodes (whichend node takes on which role is chosen in an arbitrary manner.) Allother nodes become tandem nodes as broadcast statelets reach time. So insummary, the sender node initiates a statelet broadcast pattern, thetandem nodes relay the broadcast pattern according to the SHN protocol,and the chooser node receives the broadcast pattern. When the chooserreceives an incoming statelet, then there exists a restoration path forthe failed span. The path is found by tracing the exact link-level paththat the statelet followed from the sender to reach the chooser in aprocess called reverse linking

However, in the DCPC algorithm, the objective is different from that ofthe SHN protocol. In the SHN, it is required that paths joining the endnodes of a failed span be found, in order that the span be restored. Inthe DCPC algorithm, it is required that closed paths which areeffective, preferably maximally effective, at providing PC paths, overall possible span failures, are to be found. The node roles in the DCPCalgorithm are modified to reflect that closed paths are to be generatedrather than point to point paths. In the DCPC, there are two possiblenode types: originating nodes which source and receive the stateletbroadcast pattern, and tandem nodes which relay the broadcast stateletpattern.

The originating or “cycler” node initiates the statelet broadcast, thetandem nodes relay it according to the DCPC algorithm's stateletprocessing rules, and, eventually, the statelet broadcast pattern loopsback to touch the originating node. At this point, there exists a set ofsuccessive nodes (the originating node and several tandem nodes) capableof forming a closed path in the network which the originating node canchoose to select as a closed path for pre-configuration. The closed pathis defined by following the route taken by the statelet broadcastpattern to return back to the originating node. However, the originatingnode does not immediately take the closed path indicated by the firstincoming statelet it receives. Instead, it samples the incomingstatelets and updates a record of the best received closed path eachtime it receives a new incoming statelet. When there is no furtherimprovement in the best received closed path, the originating nodeterminates the statelet broadcast and initiates the construction of asingle link-level copy of this closed path. Each initiation of astatelet broadcast by a originating node eventually results in theconstruction of a closed path (if a closed path exists).

The bulk of the processing takes place in the tandem nodes of a networkin both the SHN protocol and the DCPC algorithm. The tandem nodesdetermine what the actual shape of the statelet broadcast pattern willbe through the repeated application of the statelet broadcast ruleslocally at each node.

The basic statelet broadcast rule is quite simple; for each incomingstatelet, try to broadcast a single outgoing copy of it on a spare link,whose outgoing side is unoccupied, on each span except for the span onwhich the incoming statelet arrived. This is a simple broadcast of anincoming statelet and attempts to broadcast the statelet to the largestextent possible. Simple broadcasts of incoming statelets typically onlyhappen at the start of the originating node's initiation of the stateletbroadcast because a single incoming statelet can be broadcast to formthe precursor of multiple outgoing statelets and, as the tandem nodesrelay incoming statelets, the spare link's outgoing sides, at each node,are quickly occupied by outgoing statelets. Refer to FIG. 3 for anexample of a simple broadcast of an incoming statelet. In the example, asingle incoming statelet, s₁ arrives on the incoming side of spare portof node N. There are no other incoming statelets present at the node N,so the incoming statelet s₁ is broadcast to the largest extent possible,forming the precursor of a single outgoing derivative statelet s_(o) oneach network span connected to the node N (except the span on which theincoming statelet arrived,) as for example links L2 and L3 shown in FIG.4. The statelet on each of the links L2 and L3 are identical to eachother, but differ from s₁ in that the hopcount, numPaths and routefields in s_(o) have been updated from their values in s₁.

A modification to the simple statelet broadcast is the score basedstatelet broadcast. With the addition of this rule, an incoming stateletis broadcast to the largest extent possible prioritized by a score whichis assigned to it (the rules which evaluate the score are discussed inthe next section.) If multiple incoming statelets are present at a nodethen the statelets compete, on the basis of their scores, for a maximumstatelet broadcast. A consequence of a node's incoming statelets beingbroadcast on the basis of their scores is that if a superior stateletarrives at a node and there are no free spare links available, on one ormore spans, then the incoming statelet, on each span with no unoccupiedspare link, displaces the single outgoing statelet whose precursor hasthe worst score. An incoming statelet can be broadcast on a span in oneof two ways. The first way is by simple broadcast, which happens ifthere is a least one spare link on the span whose outgoing side isunoccupied by a statelet. The second way is by displacing an outgoingstatelet whose precursor's score is both worse than the score of theincoming statelet being considered for broadcast, and, also, the worstof the scores of all the incoming statelets which form a precursor to anoutgoing statelet on the span. If there is no spare link, on a span,whose outgoing side is unoccupied and the incoming statelet's score isnot good enough to displace an existing outgoing statelet, then theincoming statelet will not be broadcast.

This competition among the incoming statelets distributes the limitednumber of outgoing statelet positions among the incoming statelets withpreference given to those statelets with good scores. The stateletbroadcast pattern, at a node, is updated continuously as new incomingstatelets appear. However, the overall pattern has the same appearanceas if the following procedure was followed. First a list of incomingstatelets is formed, for example statelets S₁-S₅ in FIG. 5, andevaluated to assign each statelet a score. Next the list is sorted sothe statelets are ordered from best score, at the start of the list, toworst score, at the end, for example S₁-S₂-S₃-S₄-S₅. Next, as shown inFIGS. 6A-6E, the incoming statelets are broadcast, without stateletdisplacement, sequentially as each appears in the list, to the fullestextent permitted by the unoccupied outgoing side of the spare links inthe network. The result of applying this procedure is that high scoringincoming statelets are fully broadcast, medium scoring incomingstatelets are partially broadcast, and low scoring incoming stateletsmay not be broadcast at all. This is analogous to the “compete”procedure in the SHN. The final broadcast pattern reflects thepreference given to better scoring incoming statelets. S₁ is fullybroadcast to form the precursor of two outgoing statelets as shown inFIG. 6A. S₂, S₃, and S₄ are only partially broadcast to form theprecursor of one outgoing statelet as shown in FIGS. 6B-6D. S₅ is notbroadcast and forms the precursor for no outgoing statelets as shown inFIG. 6E. The final broadcast pattern is shown in FIG. 7.

As previously mentioned, each statelet belongs to an index family. Whenan incoming statelet is broadcast to form an outgoing statelet, theindex to which the incoming statelet belongs is also assigned to theoutgoing statelet. All statelets of a common index in a network can betraced back to and are derived from a single outgoing statelet, at theoriginating node, which forms the basis of that index family. Asincoming statelets are broadcast to form the precursor of multipleoutgoing statelets, it can occur that multiple incoming statelets canappear at a node that all belong to the same index family. An additionalrule, which is added to the broadcast rules of a tandem node, is thatthere can only be one outgoing statelet, which belongs to a particularindex family, on any given span. The broadcast rules are processed, aspreviously described, with preference given to those incoming stateletswith the best scores but there is an additional requirement that only asingle outgoing statelet of any given index family can exist on anyspan. The incoming statelet, which forms the precursor of this outgoingstatelet, has the best score of all incoming statelets of that indexfamily which could potentially broadcast an outgoing statelet on thatspan. This rule requires that a new incoming statelet displace apre-existing outgoing statelet of the same index family, if the newincoming statelet would form a superior precursor. This has the effectof shifting the broadcast pattern for a particular index family from theoriginal precursor statelet to the superior incoming statelet. Refer toFIGS. 8A-8C for an example of the arrival of a superior incomingstatelet s₂ where an incoming statelet s₁, of the same index family,exists. In FIG. 8A, N has a single incoming statelet s₁ belonging toindex m. In FIG. 8B, statelet s₂, also belonging to index m, arrives atnode N with a better score. As shown in FIG. 8C, s₁'s statelet broadcastis shifted to s₂. Thus, arrival of a superior incoming statelet s₂ atthe node causes the broadcast pattern to be shifted from the previouslypresent s₁ to s₂. s₁'s broadcast on s₂'s incoming span is unchangedbecause a statelet may not be broadcast on its incoming span. s₂broadcasts an outgoing statelet on s₁'s incoming span because there is aspare link available on the span.

The final broadcast rule is that an incoming statelet may not be theprecursor of an outgoing statelet on a span which connects the node,where the incoming statelet arrived, to a node which touches theincoming statelet's broadcast route. This route is stored in the routefield of the incoming statelet. The goal of the DCPC protocol is togenerate closed paths for preconfiguration by evaluating the stateletbroadcast patterns which originate and end at the originating node. Inorder that only simple closed paths be generated (that is closed pathswhich traverse each originating node only once), the statelet broadcastat the tandem nodes is limited to nodes not previously touched by theincoming statelets broadcast route. The only case where an outgoingstatelet may be broadcast to a node falling on the broadcast route iswhen a tandem node broadcasts to the originating node. This is requiredso that the statelet broadcasts may form closed paths by terminating onthe originating node. Refer to FIG. 9 for an example of the valid spansa tandem node may broadcast an incoming statelet on. An existingstatelet broadcast travels from the originating node A to tandem node Ealong route A-B-C-D-E. The incoming statelet arriving at tandem node Ecan be broadcast to nodes A, F, or G. Nodes F and G are valid broadcastsas they are tandem nodes which have not previously been touched by thestatelet broadcast pattern. Node A is a valid broadcast because it isthe originating node. Nodes B and C are not valid as they havepreviously relayed the statelet broadcast. If an outgoing statelet wassent to the originating node, the statelet broadcast pattern would forma closed path following A-B-C-D-E-A. Following this rule ensures thatthe successive notes along which a statelet is broadcast are not capableof forming a closed path that does not include the originating mode.

In summary, a tandem node's statelet broadcast rules broadcast eachincoming statelet to the largest extent warranted by the statelet'sevaluated score. If all spare links on a span are occupied by outgoingbroadcast statelets and an incoming statelet, with a good score, is ableto broadcast on the span but currently does not, then the incomingstatelet can displace the outgoing statelet whose precursor statelet hasthe worst score of all precursors for the span. An incoming statelet mayonly form the precursor for an outgoing statelet whose destination is anode which has not been previously touched by the incoming statelet'sstatelet broadcast route. The single exclusion to this rule is that anoutgoing statelet may be broadcast to the originating node. Anadditional rule is that only a single outgoing statelet of a given indexfamily may appear on a span. If multiple incoming statelets exist, whichare of a common index family and are able to be broadcast on a span,then the statelet with the best score is broadcast. These rules resultin the originating node receiving incoming statelets whose stateletbroadcast route traces out closed paths which begin and terminate at theoriginating node.

In the previous section a description of how a tandem node broadcastsincoming statelets on the basis of each statelet's evaluated score wasgiven. In this section it will be described how an incoming statelet isassigned a score.

Statelet Ranking

The DCPC protocol attempts to generate effective PC designs using closedpaths as PC elements. To generate good closed paths a score is requiredwhich reflects the quality that makes a particular closed path effectiveat providing span failures with PC paths. The score is chosen to reflectthe potential that an incoming statelet's broadcast route can form a PCclosed path with a high ratio of number of effective PC restorationpaths to number of spare links required to construct it to provide PCpaths for span restoration.

When an incoming statelet first arrives at a tandem node, the tandemnode must first evaluate the statelet's score before it can attempt tobroadcast the statelet. The statelet's score is equal to the ratio ofthe number of useful PC paths that the closed path, formed from a unionof the incoming statelet's broadcast route and an imaginary span joiningthe tandem node and the originating node, can provide and the number ofspare links which would be required to form the statelet's broadcastroute. The number of useful PC paths contained in the closed path areonly counted for real network spans and are not considered for theimaginary span which closes the closed path (unless the imaginary spanshould coincide with a real span.) The number of spare links required toform the broadcast route can also be viewed as the number of spare linksrequired to form the closed path when not considering the imaginaryspan. This score measures the potential that an incoming statelet'sbroadcast route, as evaluated locally at the tandem node, would form aneffective PC closed path if it should loop back and touch on theoriginating node.

The count of the number of PC paths which a statelet broadcast routecould provide, if it was shorted out to form a closed path, is updatedincrementally as the statelet broadcast route is relayed from tandemnode to tandem node. When a tandem node receives an incoming statelet itevaluates the number of PC paths which were added in the step betweenthe incoming statelet's originating node and itself. This number isadded to the total PC paths, as evaluated at the originating node, whichis contained in the incoming statelets numPaths field, to form a localnumPaths value for the incoming statelet. The local numPaths number isdivided by the value of the incoming statelet's hopcount field togenerate the incoming statelet's score. If the incoming statelet shouldform the precursor to an outgoing statelet, the outgoing statelet'snumPaths field is set equal to the incoming statelet's evaluated localnumPaths.

The PC path increment required to form the local path count for anincoming statelet is evaluated by examination of the statelet's routefield. If the route touched on any of the nodes which are adjacent tothe tandem node then the incoming statelet's route could provide PCpaths for the spans which connect the tandem node to adjacent nodes. Therule used to evaluate the number of PC paths which are provided to aspan connecting the tandem node to an adjacent span is quite simple. Ifa span connects the tandem node to a node, which is present in the routefield, then the statelet broadcast route could provide one or twopreconfigured paths; if the node is not present, the route can provideno paths. If the node is present, the route can provide only a single PCpath if the adjacent span is that on which the incoming statelet,belonging to the broadcast route, arrives. The route can provide two PCpaths if the span connects the tandem node to an adjacent node which ison the route but is not directly behind the tandem node on the route.The case where no PC paths are provided, occurs because there is no pathpossible in a closed path for a pair of nodes which are not both a partof the closed path. The case, where a single PC path is available,occurs when a span, which connects a pair of originating nodes directlyalong the closed path, fails because the only path possible is thatwhich connects the pair of nodes around the failed span. The case, wheretwo PC paths are available, occurs when a span, which connects a pair oforiginating nodes but is not directly on the closed path, fails becausethere are two paths possible between any pair of nodes on a closed path,and, in this case, the closed path is not broken by the failure of aspan. Therefore, unlike in the previous case, both paths can beexploited to aid in restoring the span failure. It should be noted thatthe total number of PC paths provided by a statelet broadcast route isnot what is calculated but the total number of useful PC paths. If anincoming statelet's broadcast route is evaluated and it is found tooverprovide a span with PC paths, either because the span has very fewworking links or the span's working links have been provided PC paths bypreviously formed closed paths, then only the number of paths which areuseful to restoring a span are counted. It can be determined if a PCpath would be useful in restoration because each node maintains a listof the number of uncovered working links on the spans which join it toother nodes and this list is updated as PC closed paths are formed inthe network.

The method used in the previous paragraph to evaluate the total PC pathcount for an incoming statelet is valid only when considering theincoming statelet for broadcast to another tandem node. If a statelet isbeing considered for broadcast on a span which connects the broadcastingtandem node to the originating node then the total PC count must beevaluated in a slightly different manner. The rules given in theprevious paragraph are valid except that the number of PC paths whichare available for a span, which has the originating node as an end node,can be at most one. This is because when an outgoing statelet isbroadcast to the originating node, it is closing the statelet broadcastroute to form a closed path. The span connecting the broadcasting tandemnode to the originating node falls directly beneath the closed pathformed by the broadcast route and a closed path can provide only asingle PC path to the failure of such a span. This difference requires,in effect, that a tandem node evaluate two different scores for eachincoming statelet: a score to be used for broadcast to other tandemnode, and a score to be used for broadcast to the originating node. Bothscores are still calculated by taking the ratio between the total PCpaths, as calculated locally, to the value of the incoming statelet'shopcount field. Refer to FIGS. 10A-10F and 11 for examples of how thetotal PC path count is evaluated and updated at successive tandem nodesalong a statelet broadcast route. In FIG. 10A, an incoming stateletarrives at node T with broadcast originating at originating node S. Thestatelet's broadcast route is S-1-2-T. Tandem Node T evaluates PC pathsadded for the spans joining it to adjacent nodes. Potential failed spanswhich are considered by Node T are T-2, T-1, T-3 and T-S. PC paths arefound for the closed path formed from the union of route S-1-2-T and aspan joining node T to the originating node S. As shown in FIG. 10B,there are no PC paths in case of failure of span T-3. As shown in FIG.10C, there is one PC path in case of failure of span T-2. As shown inFIG. 10D, there are two PC paths in case of failure of span T-1. Asshown in FIG. 10E, there are two PC paths when a broadcast is made to atandem node in case of failure of span T-S. As shown in FIG. 10F, thereis one PC path when a broadcast is made to the originating node in caseof failure of span T-S. In FIG. 11, a statelet broadcast reaches theoriginating node with Route S-1-2-3-4-S. As each tandem node receivesthe statelet broadcast it increments the total preconfigured path count.The total number of preconfigured paths for S-1-2-3-4-S is equal to 9(0+1+1+(1+2+2)+1+1=9). One PC path is available for the failure of aclosed path span. Two PC paths are available for a non-closed path spanwhose end nodes are originating nodes.

Initiation of Statelet Broadcast

All statelet broadcasts originate at the originating node. To initiatestatelet broadcast, the originating node places a single outgoingstatelet on an available spare link on each span falling on theoriginating node. Each outgoing statelet is assigned a unique index. Ifa span does not have an available spare link, either because the spanhas no spare links or because the span's spare links have beenpreviously used to form PC closed paths, then no outgoing statelet isplaced on it. Each outgoing statelet has its sendNode field set to thename of the originating node, its hopcount field set to a value of 1hop, and its numPaths field set to zero. Also, each outgoing statelethas its route field set to contain only the name of the originatingnode, or equivalently in the alternative manner set forth herein forestablishing the route field, all the cells in the route field are setto NIL except for the contents of the cell, corresponding to the nodewhich receives the outgoing statelet, which is set to the name of theoriginating node.

After initiation of statelet broadcast the originating node initiatesthe sampling of incoming statelets.

After the statelet broadcast is initiated, the originating node waitsand samples the incoming statelets arriving on its spare links. Eachsuccessful arrival of an incoming statelet represents a set ofsuccessive nodes capable of forming a closed path with a score attachedto that closed path. The score for each incoming statelet is found bytaking the ratio of the incoming statelet's numPaths field to that ofits hopcount field. The value is equal to the number of useful PC pathswhich the closed path can provide per spare link which would be requiredfor its construction. At the start of statelet sampling the closed pathresets a timer and then waits. As new statelet events arise on theincoming sides of the originating node's spare links, it examines thenew incoming statelet and determines its score. If the closed pathrepresented by the new statelet has a score which betters the scores ofall previously received statelets, the originating node records theclosed path and its score and resets the timer. When the timer timesout, the originating node initiates the building of the best non-zeroscoring closed path which it has received. If the originating node haseither received incoming statelets which have a score of zero or hasreceived no incoming statelets at all, the originating node does notinitiate the building of a closed path. Instead, it terminates its roleas the originating node and signal another predetermined node to assumethe role of the originating node.

This sampling of incoming statelets is markedly different from the wayreceived incoming statelets are treated by the chooser node in the SHNprotocol. Immediately upon reception of an incoming statelet, thechooser node reacts by initiating reverse linking, back along theincoming statelet's broadcast path, to form a restoration path betweenthe sender and chooser nodes. The reason for this is that a stateletbroadcast in the DCPC protocol tends to improve in its score as itprogresses unlike in the SHN where a statelet broadcast tends to worsenin score. This difference is a consequence of the different goals of thetwo protocols. The SHN uses statelet broadcasts to represent node tonode paths, between the sender and chooser nodes, and, in general, ashorter path makes more efficient use of network spare capacity than alonger path and is more reflective of the real time pressure inrestoration. Therefore, a SHN statelet broadcast's score worsens as itprogresses and becomes longer. As shown in the previous section, aclosed path tends to contain more PC paths as it grows in size and,therefore, the score of a DCPC statelet broadcast, which represents aclosed path, improves as it increases in size. This means that reactingimmediately to an incoming statelet in the DCPC protocol gives, ingeneral, a closed path which does not have a very good score. Samplingthe incoming statelets forces the originating node to give thegeneration of incoming statelets, with better scoring broadcast routes,a chance to occur. The DCPC has the luxury of permitting the sampling ofsignatures because it is being run in anticipation of a span failure,and not in response to one, and so does not require the fast real timeresponse present in the SHN.

If the originating node has found a non-zero scoring closed path in thestatelet sampling stage, it initiates the building of a single copy ofthe closed path. As mentioned in the previous section, a stateletbroadcast in the DCPC protocol improves in score as it progresses. Aconsequence of this is that there is no guarantee that the actual linklevel path, that an incoming statelet's broadcast path traversed toreach the originating node, will still be occupied by the statelet'sbroadcast. Because a statelet broadcast's score improves as itprogresses there is nothing to protect its root from being displaced byother better scoring statelet broadcasts in the tandem node broadcastsrules. At each tandem node, a statelet broadcast builds upon thestatelet broadcasts preceding it at other tandem nodes. The relativelylow scoring portion of the broadcast occurs earlier in the broadcastroute and can be displaced by the arrival of new higher scoring incomingstatelets at the preceding tandem nodes. A consequence of this is thatthe network wide statelet broadcast pattern will not eventually growstatic, unlike in the SHN protocol, but continues to fluctuate as theroots of broadcast patterns are continuously displaced by the arrival ofbetter scoring statelets. However, in the previous sampling stage, theoriginating node samples the incoming statelets and maintains a recordof the best scoring closed path. The record contains only the route theclosed path's statelet broadcast travelled to reach the originatingnode, not the actual path, but this is all that is required to build asingle copy of the closed path. It is guaranteed that the spare capacityrequired to construct a copy of the closed path will be availablebecause the closed path's statelet broadcast route was only broadcast onunoccupied spare links.

Building a Closed Path

The first step in initiating the construction of a closed path, is thatthe originating node cancels all outgoing statelets. This causes atermination of the statelet broadcasts as the originating node is thesource of all statelet broadcasts within the network. In effect,although the statelet broadcasts would continue to fluctuate if left toproceed, the originating node brings the statelet broadcasts to aconclusion. Next, the originating node examines the route field whichwas stored for the closed path. The cell corresponding to theoriginating node contains the name of the tandem node which preceded theoriginating node on the broadcast route. The originating node then scansfor the first unoccupied spare link on the span joining the originatingnode to the node indicated by the route field. The originating node thenplaces an outgoing statelet, on this spare link, with a copy of theclosed path's route field and a special reserved index family whichindicates the outgoing statelet is not a normal statelet broadcast but aclosed path construction request.

The tandem node, upon reception of the closed path construction stateletfrom the originating node, cancels all outgoing statelet broadcasts.Then it examines the route field on the construction request stateletfor the name of the tandem node which preceded it on the closed path'soriginal broadcast route. It then locates the first unoccupied sparelink on the span joining it to the node which preceded it on thebroadcast route. Next, an outgoing construction request statelet isplaced on this link, instructing the preceding route node to continueconstruction of the closed path. The tandem node's local list ofuncovered span working links is updated, to reflect the PC paths whichare available in the newly formed closed path, so that future iterationsof statelet broadcasts can be scored accurately. Also, the tandem nodeforms a crossconnection between the link on which the incomingconstruction request statelet arrived and the link on which it sent theoutgoing construction request statelet. The preceding tandem nodes onthe broadcast route reacts in a similar manner, continuing theconstruction of the closed path.

Eventually, the construction of the preconfigured closed path loops backto the originating node and an incoming closed path constructionstatelet arrives on a spare link. The originating node then forms acrossconnection between this spare link and the link on which itoriginally placed the outgoing statelet which initiated the closed pathconstruction thus completing the closed path. The originating node alsoadjust its local list of uncovered working links in a similar manner tothat of the tandem nodes. Finally, the originating node cancels alloutgoing statelets, and initiates a new statelet broadcast.

Network Level Representation of DCPC

The previous section gave a network node level description of the DCPCprotocol. In this section a network level representation of theprotocol's execution will be given. The DCPC protocol is an example of agreedy algorithm. A greedy algorithm is an algorithm which iterativelyconstructs a solution to a problem on the basis of the state of theproblem at the start of the iteration. The state of the problem ismodified, after each iteration, by the application of the iterativelygenerated solution. The DCPC protocol can be seen to be greedy becauseit iteratively generates a single closed path per iteration and theclosed path is formed on the basis of the current network configurationas represented by the number of unoccupied spare links and the number ofuncovered working links present in the network spans. As the closed pathis generated these network values are modified by the presence of PCpaths in the newly generated closed path and the utilization of sparelinks needed to construct the closed path.

Globally, the DCPC protocol executes in the following manner. Each nodein the network takes a turn at being the originating node in an orderwhich is predetermined and is stored locally within the nodes. As eachnode assumes the role of the originating node, it iteratively generatesclosed paths, using the rules outlined in the previous section, until itcan either no longer generate a closed path or the closed paths that itcan generate all receive a zero score. At this point the currentoriginating node gives up the role and signals the next node in theseries to become the originating node. The new originating nodegenerates as many closed paths as it can until it too is no longer ableto generate useful closed paths. The preconfigured closed pathsgenerated by each node alter the network's configuration as it is seenby later originating nodes. The role of the originating node issuccessively assumed by all the network nodes and when the last node hasterminated its role as the originating node, preconfigured closed pathgeneration stops.

The execution time of this protocol will be many times longer than thatof the SHN. This is because a statelet broadcast is required for eachgenerated PC closed path and because of the timer delays introduced bythe originating node's sampling of its incoming statelets. However,because the DCPC protocol is being run in anticipation of a span failureevent instead of in reaction to one like the SHN, it has the luxury ofbeing able to have a relatively long execution time.

The order in which each node assumes the role of the originating node isimportant because the originating node can only generate closed paths ofwhich it is a part. Depending on the current network configuration,choosing a certain node to assume the role of the originating node couldcause the generation of closed paths which are wasteful in the use ofthe network spare capacity. An originating node, on which spans fallwhich require minimal restoration paths, could result in closed pathswhich are larger than required or which cover spans which do not requiremany PC paths.

A node order with which this algorithm may be used is determined by thetotal number of working links falling on each network node. Each networknode has a total generated of the number of working links contained inthe spans which fall on it. The order in which the nodes assumed therole of the originating node is generated by sorting the nodes indescending order of the calculated working links total. The reverseorder is also run to evaluate any performance difference between the twoorders. The node's working link total is used to determine theoriginating node ordering since it seems reasonable that if a nodeterminates a large number of working links then it should receive anearly opportunity to form a part of the PC closed paths generated.

Simulated Implementation

As an initial evaluation of the potential of realizing an algorithm,which could generate preconfigured closed path designs in a distributedmanner, a simple simulation approach was taken. The distributed protocolwas simulated in an iterative manner with the state of the currentiteration being determined by only the state of the previous iteration.In each iteration, the outgoing statelet broadcast pattern, for eachnetwork node, is generated on the basis of the node's incoming stateletbroadcast pattern in the previous iteration. Each node then assumes itsnewly calculated broadcast pattern simultaneously and in lockstep.Successive iterations are generated in a similar manner.

This simulation method approximates the case where a network's stateletinsertion delay is large compared to the statelet processing delay.Insertion delay is due to the time required to transmit a stateletthrough a limited bandwidth communication channel while processing delayis due to the time required by the computation element present in eachnode to process incoming statelet events.

As an example of the iterative nature of the simulation, FIG. 12contains a plot of the score of the best received incoming statelet, bythe current originating node, versus the simulation iteration. In theplot, four distinct structures appear; each structure corresponds to thesearch and construction of a single PC closed path. At the left mostedge of each structure, the plot can be seen to start at zero and thenstep up. Each step corresponds to the arrival of an incoming stateletwith a superior score and an improvement to the best candidate closedpath received by the originating node. The structure then plateaus whichcorresponds to the originating node's timer counting down (the timer wasset to countdown 50 iteration.) When the timer counts out, theoriginating node initiates the construction of the closed path andresets the best received score to zero (this corresponds to the rightedge of each structure.) Although, the graph goes up to only 300iterations, the actual simulation continued for 500 more iterations.However, no more PC closed paths were possible within the network afterthe construction of the fourth closed path and, so, the plot remains atzero after this point. FIGS. 13A-13H show examples of the successivedisplacement of the best received closed path candidate as a originatingnode samples the incoming statelets generated by the evolving stateletbroadcast. The statelet broadcast, from which the incoming signaturescontaining these closed paths originate, is the broadcast represented bythe left most structure in FIG. 12. The arrival of each superiorincoming signature, can be seen as a step in the left most edge of thisstructure. During the evolution of the statelet broadcast, 8 superiorincoming statelets arrive at the originating node (superior in the sensethat their score is better than that of any previously receivedstatelet.) Each closed path in FIGS. 13A-13H corresponds to the closedpath contained within a superior incoming statelet and the score abovethe closed path corresponds to the statelet's score. The originatingnode shown in each of FIGS. 13A and 13H is shown in bold, while the boldclosed path in each figure is the best received closed path. The closedpaths are ordered in the same order of arrival as the superior stateletto which each closed path corresponds. After the arrival of the 8thsuperior statelet, no more incoming statelets with better scores appearat the originating node. Eventually, the originating node's timer timesout and the 8th closed path, as the best received closed path, is formedwithin the network's spare capacity.

FIGS. 14A-14D, and 15A-15D show how a PC pattern that is generated inNet1 by the DCPC, with originating node order sorted in order ofdecreasing total node working links and increasing total node workinglinks, respectively. Table 4 contains the network restorability results,for the decreasing order case, over all possible span failures forrestoration using only conventional KSP restoration, restoration usingonly pure PC paths, and two-step restoration. Table 6 contains therestorability results for the increasing order case. Table 5 containsthe total crossconnection events, for the decreasing order case, overall possible failed spans, for KSP restoration alone and for two-steprestoration. The crossconnection results for the increasing order caseis presented in Table 7.

FIGS. 16A-16F and 17A-17E contain examples of the PC patterns generatedin Net1 by the DCPC, for decreasing and increasing closed path order,when run within the network sparing plans generated by the 100% PCclosed path restorable IP method (IP2-closed path). Tables 8 and 10contain the restorability results for the two orders when usingIP2-closed path with the test networks. Tables 9 and 11 contain thecorresponding crossconnection results.

For the original network sparing plans, the performance of thedistributed closed path generator at times approached that of the IPbased PC closed path design (it equals the performance for Net1 usingthe descending ordering) but, as the distributed algorithm is based on aheuristic, there was no guarantee of optimal closed path generation.Although, the overall performance did not exactly equal that of the IPPC designs, the results were still quite good. Using the descendingorder, the distributed closed path generator generated PC restorabilityscores which ranged between 82% and 95%. The overall restorability usingPC with leftover KSP restoration varied between 94% and close to 100%.Also, the crossconnect results showed the result of using a simplepattern as the PC plan's building block; the total number ofcrossconnection opens was quite low for all test networks. In addition,the total number of crossconnection closures was significantly lower forrestoration using PC paths compared to KSP restoration alone, in all thetest networks.

The performance of the DCPC, when run using the sparing generated forthe test networks using IP2-Closed path, was in general better than whenrun using the original sparing, with improvements in PC restorability,two-step restorability and total crossconnection event numbers. However,it did not achieve, although it did approach, the performance of theoriginal PC closed path plan which was generated by the IP.

As expected the performance of the descending order execution was, ingeneral, better than that of the increasing order execution producingbetter PC restorability scores and PC plus leftover KSP restorabilityscores for 4 out of the 5 test networks when run in the test network'soriginal sparing plans (decreasing order was better than increasingorder in 3 of the 5 best networks when using the sparing plans generatedusing IP2-closed path.) The algorithms dependency on the order in whichthe network nodes assume the role of the originating nodes is shown bythe variation in the two orders scores.

Apparatus for Implementation

The construction and operation of apparatus for implementation is shownin FIGS. 18-21. FIG. 18 expands the view of the DCS machines withmodifications, placed at every node of the networks in FIGS. 1A, 1B and2 or any other node described in this patent document. With reference toFIG. 18, it can be seen that the creation of a pre-configured closedloop network requires two additional elements in the crossconnectionmachines of the network. In FIG. 18, components 30, 32, 34, 36, 38, 40,42 and 44 comprise the normal general embodiment of a crossconnectmachine, for the purposes that are essential here. The dashed outerboundary in FIG. 18 represents nodes. The known crossconnect machinestructure is comprised of bi-directional transmission interfaces 30,which are connected to external transmission lines 16, which may becoaxial cable, fiber, or radio, for example, and connected internally tothe switching matrix 32 of the DCS via paths 34, 36. The switchingmatrix is operated by a crosspoint operator means 38 which usuallycomprises hardware and software. As known in the art, the crossconnectswitching matrix and all interface cards are directly monitored andcontrolled exclusively by the DCS operating system 40 and matrixreconfigurations are only performed through external command andtelemetry links by a remote administrative center often called NOC(Network Operations Center).

For implementation, means 50 are added to detect statelets, addedtransparently to the carrier signals of the network and provide a methodfor directly reconfiguring the switching matrix, without remote control,in accordance with the logical method implemented by the Finite StateMachine (FSM) 70.

Each node in the network has a crossconnect machine incorporatingstatelet receiving and transmitting circuit means 50 on eachtransmission interface card 30 and an independent FSM 70 for processingreceive statelets, controlling transmit statelets and recognizingstatelet determined conditions under which the FSM requests operation ofa specific crosspoint in switching matrix 32 through shared access tothe Crosspoint Controller 38.

FIG. 19 is an expanded view of the statelet transmit and receivecircuitry shown as item 50 in FIG. 18. FIG. 19 is presented forconsideration in conjunction with FIGS. 20A-20C which detail the logicalcontent of the statelet receiving and transmitting registers andassociated status ports shown only as registers in FIG. 19.

With reference to FIG. 19, the existing receive propagation path 12 andthe existing receive signal processing interface card 30 are shown asthey were in FIG. 18 and item 50 of FIG. 18 is exploded for closer view.The existing receive signal processing circuitry 42 is connected to thenew circuits only in that a copy of the composite signal or a copycontaining only the logical binary content of the received trafficsignal plus overhead is provided as required to the statelet detectorcircuit 52. Statelet detector circuit 52 processes the received signalin a manner such that the particular bits or other attribute(s) of thesignal used for transparent conveyance of the statelet information isrecovered so as to identify the individual signaling elements of thestatelet on that signal, if any. Various techniques for doing this areknown, several of which are disclosed in U.S. Pat. No. 4,956,835. Anynumber of schemes for modulation and detection of transparent statelettransport methods are possible for implementation in transport networksusing different, transport signal formats such as DS-1, DS-3, Syntran,Sonet, or CEPT formats.

Having detected the individual signaling elements of the statelet,statelet detector circuit 52 feeds its output into a statelet receptionregister 54. Each time a complete and valid statelet is recognized by avalidity and logic control circuit 56 to be present in the stateletreception register 54 and that statelet is also seen by the logiccontrol circuit 56 to be different in from the current contents of astatelet storage register 58, then the controller 56 causes the transferof the newly received statelet in statelet reception register 54 toreceive statelet output register 58, where it will stay until anothervalid but different statelet appears on the link 18.

Once the new statelet is in the receive statelet output register, businterface 60 connected to FSM 70 (FIG. 18) raises an internal flagindicating that a new statelet has been received and, when appropriate,communicates the statelet to the controller over the bus and clears the‘new statelet’ flag once the controller has read the statelet.Thereafter, the statelet remains addressable for reference by thecontroller as may be needed in executing its control logic sequences.

Many means for accomplishing the equivalent peripheral-to-controllertransfer function are known including direct connections between eachDCS port statelet circuit and the controller or there may be a sharedinterrupt line followed by polling or a vectored interrupt arrangement,etc. In all of these schemes, however, the controller needs no memory ofits own since all statelets and status data which it needs to implementits function are retained as a distributed memory on each interface portcard of the DCS machine. The controller polls continually for newstatelets or each new statelet is explicitly brought to the controllersattention by an interrupt because the controller implementing thepre-configuration process is implemented as an event driven finite statemachine and events are determined in terms of statelet appearances,changes and disappearances.

In a preferred embodiment, when bus interface 60 is alerted that a newstatelet has been received, the bus interface logic completely handlesthe subsequent transfer of the new statelet contents and theidentification of the respective DCS interface port along with thestatus bits to the FSM 70 in FIG. 18 by known bus interfacing andtransaction techniques.

With reference to FIG. 18, the contents of any valid statelet receivedoff a transmission interface are shown and will be discussed in detail.Suffice it to say now that checkbit fields for any number of known errorchecking and/or error detecting schemes can be appended to the stateletdata field shown in FIGS. 20A-20C, and such circuitry would also beprovided in statelet reception register 54 or control logic 56 of FIG.19 without changing the nature of this embodiment.

The statelet receive and transmit circuit 50 includes a transmitstatelet storage register 88 which can be loaded by FSM 70 over bus 82and bus interface 60. This provides a means by which the FSM 70 canapply a desired transmit statelet to any desired transmission path 28 asmay be appropriate to implement the logic of the method embodied in FSM70 in FIG. 18. Through control logic 84, the bus interface can load anew statelet into Register 80 and then, once loaded, cause the repeatedcirculation of the contents of the register so that, by means of atransmit statelet modulation circuit 86, the transmit statelet inregister 80 is indefinitely repeated on the outgoing transmission link28.

Also shown in FIG. 19 is a Port Status Register 90 which stores certaininformation used by FSM 70 in the preferred embodiment. The contents ofthe port status register can be read by FSM 70 and, in the case of analarm, the alarm-detection hardware is arranged to activate businterface 60 to forward the contents portnumber register 92 and portstatus register 90 to FSM 70 without delay. Port status register 90contains elements 100, 102, 104, and 106 shown in FIG. 20. ALARM 100(FIG. 20B) is a single bit that is set if the corresponding DCStransmission interface experiences a receive loss of signal, a bit errorrate degradation, or a loss of timing, etc. or, if external transmissionterminal equipment in the same building has detected an alarm on thecorresponding channel and this information is connected over to therespective DCS interface port. Those skilled in the art of transmissionequipment and interface design will appreciate that there are manytechniques and criteria for the rapid determination of loss oftransmission integrity. The alarm bit is set by hardware and may bereset by the DCS controller after acknowledgement.

Also present in the Port Status Register is a SPARE bit 102 (FIG. 20B).This status bit is written and maintained by DCS operating System 40under conditions of normal operation. It indicates whether the givenport interface is at any given time in traffic-bearing use or is in anequipped-but-idle configuration. If SPARE is true and ALARM is false,then the given transmission interface port is available for use in themethod. If SPARE is false (the interface is carrying traffic) and ALARMbecomes true, then the method will work to restore the lost traffic bynetwork rerouting on this and all similar working links affected by thefailure event.

The CONNECTED bit 104 is not essential but is a single bit which logswhether this interface port, whether SPARE or not, is actually connectedthrough the matrix to any other interface port forming a path throughthe DCS machine. If CONNECTED is true, then the field denotedASSOC-PORT-ID 106 contains the number of the port to which this port ispresently connected through the matrix.

If CONNECTED is false, then ASSOC-PORT-ID may either contain a nulindicator or it may store information used by the FSM 70 to assist inthe faster manipulation of statelets according to the stateletprocessing method described. Specifically, when a DCS rebroadcasts astatelet arriving on a certain port to a number of other ports as in theforward flooding wave, the ASSOC-PORT register of each port that istransmitting a rebroadcast statelet stores the port number where theprecursor statelet for these repeated statelets is found. Depending oncircumstances, ASSOC-PORT registers and CONNECTED status bits can bewritten either by the normal DCS or Pre-configuration DCS controllers.

The operation of the FSM 70 is now described in relation to FIG. 21.There are two kinds of transitions within the FSM 70: conditionaltransitions are represented as dashed arrows within the FSM, whileforced transitions are represented by solid arrows. Conditionaltransitions only occur when a condition associated with the transitionis met, while forced transitions always take place. The following is alist of the FSM's conditional transitions:

Be_CyclerA command is received by a node to assume the role of thecycler node. This command can take the form of a statelet transmission,or in the case of the very first cycler to assume the cycler role, acentral mechanism or a self-scheduled alarm by the node.

CycleBuildA relayed command from the cycler node to continue thebuilding of a cycle.

RB_CyclerRelay the statelet flood which commands a node to assume therole of the cycler.

RxInt An interrupt has been raised at a port card due to the arrival ofa statelet.

NoCycle The cycler node's countdown timer has triggered and no suitablecycle was found.

FoundCycle The cycler node's countdown timer has triggered and asuitable cycle was found.

CycleBuild The construction of a cycle has looped by around to thecycler node which initiated it.

In any of the idle, sample or weight states, the FSM 70 pauses until astate transition occurs.

In the evaluate state, the FSM 70 evaluates an incoming or receivedstatelet to see if it contains a superior route. Firstly, the FSM 70makes sure the incoming statelet is of the proper type, and thencompares the score of that incoming statelet to the best previouslyfound score for statelets of that type.

In the InitCycler state, the local node at which the FSM 70 is locatedassumes the role of the originating node. The FSM 70 initializes astatelet broadcast with a single outgoing statelet on each span whichhas on it at least one available spare link. Each outgoing broadcast hasa unique index associated with it. The FSM 70 resets the current valueof a register containing the identity of the best closed path and bestclosed path store in anticipation of starting sampling. The FSM 70 setsa countdown timer alarm to the predetermined sampling interval.

In the StartCycle state, the FSM 70 has finished sampling the routesfollowed by incoming statelets, and begins to build a new closed path ofcross-connections and spare lengths. Firstly, the FSM 70 cancels alloutgoing broadcasts, and begins construction of the closed path. The FSM70 finds the node or span which was last traversed by the best scoreincoming statelet. The FSM 70 then sends a closed path constructionrequest statelet backwards along the route followed by the statelet withthe best score.

In the FinishCycle state, the FSM 70 confirms that a closed path buildrequest has looped around, and marks the port, on which the constructionstatelet arrived, as used. The FSM 70 then closes a crosspoint tocomplete the closed path, and updates the local lists of unprotectedworking links and available spares.

In the state ProcessStatelet, the node processes a new statelet. The FSM70 checks to see whether an interrupt represents a new statelet or astatelet disappearing, resets the local scores for the port, and cancelsany broadcast. The FSM 70 checks for outgoing statelets which were beingre-broadcast from a statelet which previously existed on the port, butwas overwritten by the new statelet. If such outgoing statelets exist,they are cancelled. If a statelet incoming to the node is eligible, itis re-broadcast. The FSM 70 forms a list of all incoming statelets atthe local node. It re-broadcasts each statelet to the largest extentpossible, within the existing outgoing statelet broadcast pattern.

In the state RelayBecomeCycler, the FSM 70 cancels broadcasts, findswhich node initiated a broadcast instructing nodes to become anoriginating node, and finds which node is next to become an originatingnode. The node floods or relays a copy of the command instructing thenodes to become originating nodes to all adjacent nodes except the nodefrom which the instructing statelet came.

In the ContinueCycle state, the FSM 70 first marks the span on which anincoming statelet arrived as used, cancels all broadcasts, and continuesbuilding the closed path by sending an instruction to the next nodealong the route to continue construction of the closed path. The FSM 70adjusts available spares and uncovered working spans. Finally, the FSM70 makes a cross point to form the local nodes part of the closed path.

In the CyclerDone state, the FSM 70 records that the local node has beenan originating node. If the local node is not the last originating node,it initiates a flood command which instructs the next node in apre-determined list of originating nodes to become the next originatingnode, and also marks any statelet broadcast with the ID of the localnode as the send node in that statelet. If the local node is the lastoriginating node, execution of the protocol ceases.

The method of broadcasting a statelet with the statelet carrying its ownroute information in a data field may also be used to find and createroutes of connected spare links. As in the SHN technique, when theobject is only to find a route, and not a closed route, the statelet maybe given a target node identification and it will be broadcast until itreaches the target node. The route information in the statelet may beused to find the route taken by the statelet. In this method, stateletsmay be ordered for preferential broadcast by rules that do not relate tothe numPaths field, but which may, for example, simply depend on the hopcount (the statelet with the smallest hop count being broadcast forexample). Further, in the general application of the method, theordering for preferential broadcast is not restricted to an evaluationof the numPaths and hopcount fields, but may also use other measures.

When a span failure occurs, as for example shown in FIG. 1A or FIG. 1B,traffic may be re-routed by directing the traffic along the closed pathby operating the cross-connections at the end nodes, using thecross-point controller 38 under control of the control system 40.

A pseudo-code representation of the operation of the FSM 70 follows asan appendix.

Immaterial modifications may be made to the embodiments described herewithout departing from what is covered by the claims.

TABLE 1 Restorability using KSP and IP-1 PC Cycle Design KSP Cycle-PCTwo-Step Network Restorability (%) Restorability (%) Restorability (%)Net1 100 93.66 93.66 Net2 100 96.58 100 Net3 100 96.86 99.95 Net4 10075.88 98.91 Net5 100 94.93 99.95

TABLE 2 Total Crossconnect Events for IP-PC Cycle-Oriented Designs XptsClosed Xpts Closed Xpts Opened Total Xpt Network KSP Cycle-PC Cycle-PCEvents two-step IP-1: Net1 310 0 174 174 IP-1: Net2 3984 92 2260 2352IP-1: Net3 13787 366 7086 7452 IP-1: Net4 84389 20032 55419 75451 IP-1:Net5 8529 312 4067 4379 IP-2: Net1 291 0 190 190 IP-2: Net2 4176 0 21082108 IP-2: Net3 20744 0 6860 6860 IP-2: Net4 86048 0 45216 45216 IP-2:Net5 13727 0 3850 3850

TABLE 3 Spare Capacity Comparison of Cycle- Oriented PC and KSP Designs% Excess between Network Original and PC Cycle Net1 9.09 Net2 3.07 Net30   Net4 18.54* Net5 0*   *Best feasible solution prior to prematureCPLEX termination due to memory limitation.

TABLE 4 Network Restorability using Standard KSP and PC Restorationusing the DPCD with Cycler Order in Descending Total Node Working LinksOrder KSP PC Two-step Network Restorability (%) Restorability (%)Restorability (%) Net1 100 93.66 93.66 Net2 96.87 95.01 99.43 Net3 10081.73 94.53 Net4 100 85.12 94.93 Net5 100 92.20 99.50

TABLE 5 Total Network Crossconnect Events Results using Standard KSP andPC Restoration using the DPCD with Cycler Order in Descending Total NodeWorking Links Order Xpts Closed Xpts Closed Xpts Opened Total XptNetwork KSP PC PC Events PC Net1 310 0 174 174 Net2 3984 122 2220 2342Net3 13787 1917 7011 8928 Net4 84389 8269 43277 51546 Net5 8529 455 39854440

TABLE 6 Network Restorability using Standard KSP and PC Restorationusing the DPCD with Cycler Order in Increasing Total Node Working LinksOrder KSP PC Two-step Network Restorability (%) Restorability (%)Restorability (%) Net1 100 84.51 84.51 Net2 96.87 85.33 96.87 Net3 10082.86 96.29 Net4 100 84.91 94.57 Net5 100 86.13 98.86

TABLE 7 Total Network Crossconnect Events Results using Standard KSP andPC Restoration using the DPCD with Cycler Order In Increasing Total NodeWorking Links Order Xpts Closed Xpts Closed Xpts Opened Total XptNetwork KSP PC PC Events PC Net1 310 0 162 162 Net2 3984 420 2278 2698Net3 13787 1666 6595 8261 Net4 84389 7812 44074 51886 Net5 8529 10744190 5264

TABLE 8 Network Restorability using Standard KSP and PC Restorationusing the DPCD with Cycler Order In Descending Total Node Working LinksOrder with the Sparing Plans generated by IP2-Cycle KSP PC Two-stepNetwork Restorability (%) Restorability (%) Restorability (%) Net1 10094.37 97.18 Net2 100 99.15 100 Net3 100 90.02 97.12 Net4 100 96.84 99.00Net5 100 92.42 96.44

TABLE 9 Total Network Crossconnect Events Results using Standard KSP andPC Restoration using the DPCD with Cycler Order in Descending Total NodeWorking Links Order with the Sparing Plans generated by IP2-Cycle XptsClosed Xpts Closed Xpts Opened Total Xpt Network KSP PC PC Events PCNet1 291 9 203 212 Net2 4176 20 2080 2100 Net3 20744 1074 6783 7857 Net486048 1762 42292 44054 Net5 13727 382 3828 4210

TABLE 10 Network Restorability using Standard KSP and PC Restorationusing the DPCD with Cycler Order in Increasing Total Node Working LinksOrder with the Sparing Plans generated by IP2-Cycle KSP PC Two-stepNetwork Restorability (%) Restorability (%) Restorability (%) Net1 10083.10 95.07 Net2 100 98.72 100 Net3 100 94.44 97.94 Net4 100 96.23 98.22Net5 100 86.86 97.49

TABLE 11 Total Network Crossconnect Events Results using Standard KSPand PC Restoration using the DPCD with Cycler Order In Increasing TotalNode Working Links Order with the Sparing Plane generated by IP2-CycleXpts Closed Xpts Closed Xpts Opened Total Xpts Network KSP PC PC EventsPC Net1 291 44 199 243 Net2 4176 32 2120 2152 Net3 20744 558 6836 7394Net4 86048 2222 42020 44242 Net5 13727 2044 4757 6801

DCPC FSM State Transitions

There are two kinds of transitions within the FSM in FIG. 1: conditionaltransitions are represented as dashed arrows within the FSM, whileforced transitions are represented by solid arrows. Conditionaltransitions only occur when a condition associated with the transitionis met, while forced transitions always take place. The following is alist of the FSM's conditional transitions:

Be_Cycler A command is received by a node to assume the role of thecycler node. This command can take the form of a statelet transmission,or in the case of the very first cycler to assume the cycler role, acentral mechanism or a self-scheduled alarm by the node. CycleBuild Arelayed command from the cycler node to continue the building of acycle. RB_Cycler Relay the statelet flood which commands a node toassume the role of the cycler. RxInt An interrupt has been raised at aport card due to the arrival of a statelet. NoCycle The cycler node'scountdown timer has triggered and no suitable cycle was found.FoundCycle  The cycler node's countdown timer has triggered and asuitable cycle was found. CycleBuild  The construction of a cycle haslooped by around to the cycler node which initiated it.

DCPC FSM State and Function Logic

state Idle( ) {  Pause until a State Transition occurs ; } state Sample() {  Pause until a State Transition occurs ; } state Wait( ) {  Pauseuntil a State Transition occurs ; } state Evaluate(PORT port) {  /*Evaluate new statelet to see if it contains a superior cycle */  /* Makesure it is of the proper type */  if ( port.rxReg.sendNode==nodeID) ) {  paths = port.rxReg.numPaths;   hops = port.rxReg.hopcount;   score =paths / hops;   /* Compare to best previously found cycle */   if(score > bestCycleScore) {    bestCycleScore = score;    bestCycleHops =hops;    bestCycle = copy(port.rxReg.croute);   }  } } state InitCycler() {  /* The local node is to assume the role of the cycler */ Initialize a statelet broadcast with a single outgoing statelet on eachspan which has an   at least one available spare link. Each outgoingbroadcast has a unique index associated with   it ;  Reset the currentvalue of the bestCycle and bestCycleScore in anticipation of starting  sampling. Set a countdown timer alarm to the predetermined cyclersampling interval; state StartCycle( ) {  /* Finished sampling cycles,now its time to start building one */  /* First cancel all outgoingbroadcasts */  cancelBroadcasts( );  /* Now start the construction ofthe cycle */  /* Find the node/span which was previous to the cyclerback along the cycle */  node = One node back from cycler node alongbestCycle;  span = Adjacent span which connects nodeID to node;  /* Senda Cycle construction request statelet backwards along the cycle */ cycleStartLink = link = findFreeLink(span);  cycleStartSpan = span;  /*Setup port's transmission register */  port = Port on span,link ; port.portStat=USED_SPARE; /* Set port status to show its link is beingused in a cycle */  port.txReg.hopcount = bestCycleHops; port.txReg.sendNode = nodeID;  port.txReg.croute = copy of bestCycle ; Interupt port ; } state FinishCycle( ) {  port = Port oninterupt_span,interupt_link ;  /* Confirm that the cycle build requesthas looped around */  if ( port.sendNode == nodeID && port.portStat ==AVAILABLE_SPARE ) {   /* Mark the port as used */  port.portStat=USED_SPARE;   /* Close an xpt to close the cycle */  Message Crossconnect controller to close a crosspoint betweeninterupt_span,interupt_link    and cycleStartSpan,cycleStartLink ;   /*Update the local lists of unprotected working links and available spares*/   availableSpare[cycleStartSpan]−−;  availableSpare[interupt_span]−−;  updateUnprotectedWorkers(bestCycle);  } } state ProcessStatelet( ) { /* Process the new statelet */  port = Port oninterupt_span,interupt_link ;  /* Check if interupt represents a newstatelet or a statelet disappearing */  if (port.rxReg.index==NILL) {/*Disappeared */   /* Reset the local scores for the port */  inPaths[port].paths = inPaths[port].cpaths = NILL;  inPaths[port].score = inPaths[port].cscore = NILL;  cancelBroadcastFromPort (port);  } else { /* New statelet */   calcInpaths (port);    Check for outgoing statelets which were beingrebroadcast from a statelet which     previously existed on port but wasoverwritten by the new statelet. If such outgoing    statelets existcancel them ;   /* If the statelet is eligible, rebroadcast it */   if(port.rxReg.hopcount < maxHops) {    broadcastStatelet (port);   }  Form a list of all incoming statelets on the local node. Rebroadcasteach statelet to the    largest extent possible within the existingoutging statelet broadcast pattern.  } } state RelayBecomeCycler( ) { port = Port on interupt_span,interupt_link ;  cancelBroadcasts ( );  /*Find which node initiated the becomeCycler broadcast and   which node isto next become the cycler node */  nextCycler = nextCycler indicated byport;  sendNode = sendNode indicated by port;  Flood/relay a copy of thebecomeCycler command to all adjacent nodes   except the node connectedto port ; } state ContinueCycle( ) {  port = Port oninterupt_span,interupt_link ;  /* First mark the span on which the cyclearrived as used */  port.portStat=USED_SPARE;  cancelBroadcasts ( ); node = Node which is previous to local node along port.rxReg.croute ; span = Adjacent span connecting local node to node ;  link =findFreeLink(span);  nextPort = Port on span,link ;  /* Continuebuilding the cycle */  nextPort.portStat=USED_SPARE; nextPort.txReg.index = port.rxReg.index;  nextPort.txReg.hopcount =port.rxReg.hopcount;  nextPort.txReg.sendNode = port.rxReg.sendNode; nextPort.txReg.croute = Copy of port.rxReg.croute;  Interupt nextPortto transmit a continue Cycle construction statelet;  /* Adjust availablespares and uncovered working */  availableSpare[span]−−; availableSpare[interupt_span]−−;  updateUnprotectedWorkers(port.rxReg.croute );  /* Make an xpt to form the local node's part ofthe cycle */  Message Crossconnect controller to close a crosspointbetween interupt_span,interupt_link   and span,link ; } stateCyclerDone( ) {  cancelBroadcasts( );  Record that the local node hasbeen the cycler;  if (Local node is not the last cycler) {   /*Initialize the next cycler commnd */   Initiate a Flood commandindicating the locally stored nextCycler node become the next cycler   and, also, mark the flood with the local nodeID as the sendNode ;  }else {   /* The local node is the last cycler node and execution of the   DCPC protocol may cease. */   Stop running the DCPC protocol;  } }function findFreeLink (SPAN span) {  /* Find the first available sparelink on the span */  Find a spare link on span, which has not beenpreviously used   in forming a cycle, if possible ; } functioncancelBroadcasts ( ) {  /* Cancel all outgoing statelet transmissions */ Cancel all outgoing statelet broadcasts on all ports in the local node; } function updateUnprotectedWorkers (CYCLE cycle) {  /* Update thelocal nodes list of unprotected working links (links which do not havepaths   available to them within the preformed cycles present in networkspare capacity) to account   for the placement of a single copy of cycle*/  /* Trace through cycle, finding adjNodes */  for (i = index from thestart to the end of the route contained in cycle) {   if (the nodecontained in cycle[i] is adjacent to the local node) {    if ( the nodecycle[i] appears directly before or after the local node in cycle ) {    adjSpan = Adjacent span connecting the local node to the node incycle[i] ;     uncoveredWorking[adjSpan]−−;  /* Can only get one path ifadjacent along the cycle */    } else {     adjSpan = Adjacent spanconnecting the local node to the node in cycle[i] ;    uncoveredWorking[adjSpan]−=2; /* Get two paths if not adjacent alongthe cycle */    }    /* Check if the span has been overprovided withrestoration paths */    if (uncoveredWorking[adjSpan] < 0)    uncoveredWorking[adjSpan] = 0;   }  } } function calcInpaths (PORTport ) {  /* Calculate the score and number of restoration paths for anew incoming statelet */  /* Get some info out of the port card receiveregisters */  cinpaths = inpaths = port.rxReg.numPaths;  cycle =port.rxReg.croute;  hops = port.rxReg.hopcount;  cycler =port.rxReg.sendNode;  /* Find out if you're next to the cycler */ byCycler = TRUE if the local node is adjacent to the cycler in thenetwork -- FALSE otherwise ;  /* Go through and check the cycle for thePC paths it may provide */  /* For the case when the statelet is to betransmitted to a tandem node */  for (i= From the start to the end ofcycle) {   if (the node in cycle[i] is adjacent to the local node ) {   if (the node cycle[i] is directly before the local node in thebroadcast     of the statelet in port) {     npaths = 1; /* Can providea single path */    } else {     npaths = 2; /* Can provide a pair ofpaths */    }    /* Find an updated numPaths to account for thestatelet's latest hop */    /* Account for the overprovision of a spenwith restoration paths;     only count paths useful to restoration */   if ( npaths < uncoveredWorking[j] ) {     inpaths += npaths;    }else {     inpaths += uncoveredWorking[j];    }   }  }  /* Go throughand check the cycle for the PC paths it may provide */  /* Fortransmission to the cycler node */  if (byCycler) {   for (i= From thestart to the end of cycle) {    if (the node in cycle[i] is adjacent tothe local node ) {     if ( the node cycle[i] is directly before thelocal node in the broadcast of the       statelet in port OR the nodecycle[i] is the cycler node ) {      npaths = 1; /* Can provide a singlepath */     } else {      npaths = 2; /* Can provide a pair of paths */    }      if ( npaths < uncoveredWorking[j] ) {       cinpaths +=npaths;      } else {       cinpaths += uncoverdWorking[j];     }    }  }  }  /* Store the results */  inPaths[port].paths = inpaths; inPaths[Port].cpaths = cinpaths;  inPaths[port].score = inpaths / hops; inPaths[port].cscore = cinpaths / hops; } function broadcastStatelet(PORT port) {  /* Broadcast a single incoming statelet to the largestdegree possible within an existing   broadcast pattern */  span = Spanon which port exists ;  link = Link on which link exists ;  cycler =port.rxReg.sendNode;  cycle = port.rxReg.croute;  hops =port-rxReg.hopcount;  index = port-rxReg.index;  /* Attempt torebroadcast port's incoming statelet on each span falling on the localnode */  for (span_i= From start to end of set of spans which fall onthe local node) {   cycleNode = FALSE;   broadcast = TRUE;   if ( (theadjacent node on the other side of span_i is not in cycle OR     theadjacent node on the other side of span_i is the cycler) AND     (span_i1 = span) ) {    /* Start competing with the pre-existing statelets fora place */    worstScore = INFINITY;    index_present = FALSE;   betterPrec = FALSE;    bestPrecLink = unoccupiedLink = worstLink =NILL;    for (link_j= From start to end of set of spare links on span_i){     port_out = Port on span_i,link_j ;     if (port_out.portStat==AVAILABLE_SPARE ) {      if ( port_out.txReg.index ==NILL ) { /* No statelet present */       if (unoccupiedLink==NILL)       unoccupiedLink=link_j;      } else {       p_span = Span on whichthe incoming statelet from which           the statelet on port_out wasrebroadcast ;       p_link = Link on which the incoming statelet fromwhich           the statelet on port_out was rebroadcast ;       port_in= Port on p_span,P_link ;       if (index==port_in.txReg.index) { /*Statelet with matching index exists */        index_present = TRUE;       /* Check if the new statelet would is a better statelet than thaton port_in */        if (the cycler node is the adjacent node across onspan_i ) {         if (inPaths[port].cscore > inPaths[port_in].cscore) {         betterPrec = TRUE;          bestPrecLink = link_j;         }       } else {         if (inPaths[port].score >inPaths[port_in].score) {          betterPrec = TRUE;         bestPrecLink = link_j;         }        }        if(betterPrec) {         Have found a link to broadcast on as there canonly be a single statelet of a          given index on a span - Stopsearching this span by Breaking out of loop.        }       } else {       /* Check if the new statelet has a better score than the existingstatelet*/        if (cycler==adjNode[i]) {         if ((inPaths[port_in].cscore < inPaths[port].cscore) &&         (inPaths[port_in].cscore < worstScore) ) {          worsstLink= link_j;          worstScore = inPaths[port].cscore;         }        }else {         if ( (inPaths[port_in].score < inPaths[port].score) &&         (inPaths[port_in].score < worstScore) ) {          worstLink =link_j;          worstScore = inPaths[port_in].score;         }        }      }      }     }    }    /* Broadcast the statelet on a link on thespan (if possible) */    out_link = NILL;    if (|index_present) {    if (unoccupiedLink!=NILL) {      out_link = unoccupiedLink; /*Broadcast on an unoccupied link */     } else {      if(worstLink!=NILL) {       out_link = worstLink; /* Otherwise, broadcaston the worst link */      }     }    } else {     if (betterPrec)     out_link = bestPrecLink;    }    if (out_link!=NILL) {     /* Tx anoutgoing statelet */     port_tx = Port on span_i,out_link;    port_tx.txReg.index = index;     port_tx.txReg.hopcount = hops + 1;    if (the cycler on is on other end of span_i)     port_tx.txReg.numPaths = inPaths[port].cpaths;     else     port_tx.txReg.numPaths = inPaths[port].paths;    port_tx.txReg.croute = Copy of cycle;     Append local node ID Tothe end of port_tx.txReg.croute;     port_tx.txReg.sendNode = cycler ;    Interupt port_tx to indicate a new outgoing statelet is to bebroadcast ;    }   }  } } function cancelBroadcastFromPort ( int span,int link ) {  Cancel all outgoing broadcasts which originate from anincoming statelet on port ; }

1-53. (canceled)
 54. A method of operating a telecommunications networkin which the telecommunications network includes plural distinct nodesinterconnected by plural distinct spans, each span of the pluraldistinct spans including working and spare parallel links betweenadjacent nodes, and each node having a switching machine for making andbreaking connections between channels in adjacent spans, the methodcomprising: connecting spare capacity in closed paths, each closed pathextending through a set of nodes, with at least one spare link betweeneach pair of adjacent nodes in the closed path, in readiness for a spanfailure; and configuring the telecommunications network to: (A) restoreworking traffic affected by a span failure on a span that is part of aclosed path by routing signals along surviving nodes of the closed path;and (B) restore working traffic affected by a span failure on a spanbetween two nodes that are part of a closed path, and the span is notwithin the closed path, by routing signals along the closed path.
 55. Amethod of operating a telecommunications network in which thetelecommunications network includes plural distinct nodes interconnectedby plural distinct spans each node having a switching machine for makingand breaking connections between channels in adjacent spans, the methodcomprising: connecting spare capacity in closed path self-healing rings,each closed path self-healing ring extending through a set of nodes,with at least one spare link between each pair of adjacent nodes in theclosed path self-healing ring, in readiness for a span failure; andconfiguring the telecommunications network to: (A) restore workingtraffic affected by a span failure on a span that is part of a closedpath self-healing ring by routing signals along surviving nodes of theclosed path self-healing ring; and (B) restore working traffic affectedby a span failure on a span between two nodes that are part of a closedpath self-healing ring, and the span is not within the closed pathself-healing ring, by routing signals along the closed path self-healingring, thereby enabling the restoration path of a self-healing ringnetwork while retaining the capacity efficiency of a span restorablemesh network.
 56. The method of claim 55 wherein restoration protocolrequires only the end nodes of a failed span to act to substitutetraffic with no signaling requirement to or amongst other network nodesto restore traffic flow.
 57. The method of claim 55 wherein tworestoration paths per failure contribute to restoration of any networkspan failure including failures not on the closed path.
 58. A method ofoperating a telecommunications network in which the telecommunicationsnetwork includes plural distinct nodes interconnected by plural distinctspans, each node having a cross-connect switch for making and breakingconnections between links in adjacent spans forming span paths throughthe node, the method comprising: (a) providing a set of successive nodescapable of forming a path in the network, with at least one spare linkbetween each pair of adjacent nodes in the path; (b) transmittingbetween network nodes state messages that convey information related tonetwork connectivity; (c) receiving at least one of said incoming statemessages at a network node, and transmitting at least one of said statemessages from that node to one or more nodes adjacent to that node andconnected to that node by at least one spare link; (d) before any spanfailure, based on said connectivity information distributed via saidstate messages, determining one or more restoration routes betweenselected network nodes to be used to carry communications traffic; and(e) upon detection of a span failure, routing affected communicationtraffic over said one or more predetermined restoration routes.
 59. Themethod of claim 58 wherein each state message identifies the originatingnode.
 60. The method of claim 58 wherein selection of restoration routesis based on connectivity information and one or more optimizationcriteria.
 61. The method of claim 60 wherein the optimization attemptsto minimize the number of unprotected spans in the network.
 62. Themethod of claim 60 wherein the optimization attempts to minimize thetotal cost of network spans.
 63. The method of claim 58 wherein one ofthe selected network nodes in one or more restoration routes chooses therestoration path to be used between the selected nodes.
 64. The methodof claim 63 wherein the node that chooses the restoration path to beused between the selected nodes notifies other nodes along the selectedrestoration path to alter connections to permit traffic flow along theselected restoration path.
 65. The method of claim 58 wherein therestoration is carried out for a mesh type network.